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In this study, researchers developed a survey to determine student perceptions of readiness prior 
to entering an educational leadership program. The researchers analyzed and established the 
reliability and validity of the survey created to understand student readiness as they enter the 
program. The information garnered from this survey will help faculty in the program make 
instructional decisions based on the student feedback and will be used to provide suggestions to 
the faculty for program improvement. The results will also be used to contribute to program 
accreditation through the Council for Accreditation of Educator Preparation (CAEP). 

This study identified how trustworthy and dependable the instrument is in determining 
student perceptions of readiness for educational leadership based on present knowledge levels. A 
panel of experts from areas outside of the research population was used to establish content and 
face validity of the instrument. Internal consistency and reliability was measured using 
Cronbach’s alpha. Fifty-one survey scores were used from studen ts enrolled in the educational 
leadership program across the Commonwealth of Virginia. The survey was given in July of this 
year. 
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The role of a school principal has drastically changed over the past 15 years. As educational 
leadership preparation programs work to provide the timeliest and most relevant infonnation to 
their students, it is important for faculty to understand how well prepared our students are as they 
enter the program. In order to best meet our students’ needs, faculty must understand the 
students’ levels of competency in a variety of areas related to the area of educational leadership. 
Knowing students’ needs at the beginning of their Educational Leadership Program will allow 
professors in those programs to tailor instruction accordingly so that students exit programs with 
full mastery of state and national standards in the area of educational leadership. To this end, one 
of the most effective ways to gain this information is through the use of a valid and reliable 
survey. 

In addition, having baseline data of students’ perceptions of knowledge will allow faculty 
in the program to track and measure student growth as they exit the program. Pre and post¬ 
assessment of student perceptions is critical when working to improve program content and 
offerings in the future. 

The process the researchers underwent in this study will be useful to both professors and 
students in other programs of educational leadership as it will provide a concrete example of how 
to tailor course curriculum to the expressed needs of the students, much like doctors diagnose 
maladies and provide treatments specific to patients’ needs. The infonnation gained from the 
study will also help the program meet the continuous improvement expectations of the Council 
for Accreditation of Educator Preparation (CAEP), as explained below. 

Literature Review 


Definition of terms 
Reliability 

Reliability refers to the consistency of a measure or instrument. If we attain the same 
result repeatedly the measure is considered reliable. For example, “if an assessment is designed 
to measure a trait (such as introversion), then each time the assessment is administered to a 
subject, the results should be approximately the same” (Cherry, 2013, p. 1). 

Face Validity 

Anastasi (1988) defines face validity “.. pertains to whether the test "looks valid" to the 
examinees who take it, the administrative personnel who decide on its use, and other technically 
untrained observers (p.144)." 

Content Validity 

Clause (2015) defines content validity as “...how accurately an assessment or 
measurement tool taps into the various aspects of the specific construct in question. In other 
words, do the questions really assess the construct in question, or are the responses by the person 
answering the questions influenced by other factors?” 

Cronbach’s Alpha 

“Cronbach’s alpha is a model of internal consistency reliability based on the average 
inter-item correlation of an instrument” (Rovai, Baker, and Ponton, 2014, p. 545). 

Current Literature 

One of the primary purposes for starting this study was to find ways to demonstrate the 
educational leadership program is meeting the needs of the students as evidenced through student 


24 



perceptions and demonstrated growth. This information is not only useful to the faculty as a 
whole, but is also a requirement of national accrediting agencies such as the Council for 
Accreditation of Educator Preparation (CAEP). One of the primary focuses of CAEP is 
demonstrated student growth; CAEP wants to ensure programs it accredits "... advance 
excellence in educator preparation through evidence-based accreditation that assures quality and 
supports continuous improvement to strengthen P-12 student learning.” (CAEP, 2015). 

CAEP (2015) has advanced standards for Educational Leadership preparation programs. 
Three CAEP standards buttress the importance of this study. Pertinent CAEP “Advanced 
Program Components” are as follows: 

• Satisfaction of Completers 4.4: The provider demonstrates, using measures that result in 
valid and reliable data, that advanced program completers perceive their preparation as 
relevant to the responsibilities they confront on the job, and that the preparation was 
effective. 

• Quality and Strategic Evaluation 5.1: The provider’s quality assurance system is 
comprised of multiple measures that can monitor advanced program candidate progress, 
advanced completer achievements, and provider operational effectiveness. Evidence 
demonstrates that the provider satisfies all CAEP standards. 

• Continuous Improvement 5.3: The provider regularly and systematically assesses 
performance against its goals and relevant standards, tracks results over time, tests 
innovations and the effects of selection criteria on subsequent progress and completion, 
and uses results to improve program elements and processes (CAEP, 2015). 

In order to measure the satisfaction of those who complete an educational leadership 
program, it is helpful to know students’ perceptions of readiness at the beginning of the program. 
But few studies have been done to measure the satisfaction of completers of educational 
leadership programs. Orr & Orphanos (2011) found that little is known about the impact of 
innovative programs and their components on principal behavior, and most important, on how 
those behaviors influence teaching and learning. Moreover, for the research that does exist, 
“evidence” is commonly based upon the self-reported perceptions of principals or the 
perceptions of various school stakeholders rather than measurable data of school and student 
outcomes (Darling-Hammond et ah, 2007). 

A study completed by the Stanford Educational Leadership Institute examined several 
kinds of evidence about program outcomes: candidates’ and graduates’ perceptions about their 
preparedness for various aspects of the principalship, self-reports of practices in key areas known 
to be related to effectiveness, and entry and plans to remain in the principalship, compared to a 
national sample; perceptions of employers about graduates’ capacities; observations of 
graduates’ practices on the job; and data about student achievement trajectories in graduates’ 
schools, but did not study students’ satisfaction with the educational leadership program 
preparedness. (Davis & Darling-Hammond, 2012). In “Student Satisfaction of Online Courses 
for Educational Leadership” authors Pauline Sampson, John Leonard, Julia Ballenger, and Craig 
Coleman (2009) examined students’ satisfaction of online courses in a principal and 
superintendent certification program in the online Educational Leadership program at Stephen F. 
Austin University. The study “explored the students’ satisfaction of course components: 
instruction, communication, assessment, leadership, teamwork, professionalism, and 
respect/diversity. The most recent group of students (2009) with a totally online delivery fonnat 
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completed the survey and showed an overall positive satisfaction with overall means between 
3.77 and 4.30 on a five point Likert-scale with a 5 meaning strong agreement with satisfaction.” 

In March of 2008, Douglas Summer, a doctoral student at Baker University completed 
his dissertation entitled “A Measurement Of Student Satisfaction Levels As A Means Of 
Program Evaluation: An Examination Of Baker University’s Educational Leadership Doctoral 
Program,” concluding, “there was general satisfaction expressed by the participants in the study 
across all of the program design but... a need for improved efforts in the area of student 
advising.” 

More has been written about the quality and strategic evaluation of educational leadership 
programs but not necessarily as those program evaluations pertain to the satisfaction of program 
completers. Most notably the UCEA Center for the Evaluation of Educational 
Leadership Preparation and Practice (CELP, 2015) “...makes available valid and reliable 
evaluation research tools, methods and training materials and strategies for leadership 
preparation programs. Through this center, UCEA (2015) fosters: 

1. The collection and analysis of survey evaluation research data for program benchmarking 
and in-depth multi-program analysis of program features and graduate career and 
leadership practices outcomes. 

2. The creation of a systematic process for collecting and analyzing state data on degrees 
and certification by institution, and career advancement and school progress by graduates 
and institutions. 

3. The provision of evaluation training provides technical assistance and support for 
leadership preparation programs and establishes regional train-the-trainer opportunities to 
increase evaluation technical assistance capacity locally. 

4. The creation of a sustainable system for on-going evaluation research to support the on¬ 
going evaluation and improvement of leadership preparation programs.” 

Similarly, many educational leadership programs strive for continuous improvement but 
there is little or no literature to verify that such improvement has been based upon student 
perceptions of readiness at the beginning of a program. Rather, improvements have been founded 
on principles set by national and state standards. In 1996 the Council of Chief State School 
Officers promulgated the Interstate School Leaders Licensure Consortium (ISLLC standards); 
these standards have been updated several times over the past nineteen years, the latest update is 
currently in progress. Those six standards specify attributes and qualities school leaders 
throughout the nation must possess and therefore, form the basis for most educational leadership 
preparation programs in general. It is significant to note that Chief State School Officers 
established these standards as desirable attributes of school administrators throughout the nation- 
-not in response to students’ perceptions. 

The ISLLC standards were followed by the Educational Leadership Consortium Council 
Standards (ELCC) in 2011. These standards specify attributes for “institutions undergoing 
NCATE Accreditation and ELCC Program Review for Advanced Programs at the Master, 
Specialist, or Doctoral level that prepare Assistant Principals, Principals, Curriculum Directors, 
Supervisors, and other educational leaders in a school building environment.” Accordingly, 
educational leadership programs follow those seven standards to produce graduates whose skills 
are in compliance with those national standards and the National Council for Accreditation of 
Teacher Education, but not in response to students’ perceptions of educational leadership 
preparation programs. 
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Another example of a national organization setting standards that educational leadership 
programs throughout the nation pay heed to is the Quality Principles for Educational Leadership 
Programs specified by the Teacher Education Accreditation Council (TEAC, 2014). “TEAC’s 
principles and standards are compatible with the standards promulgated by many states and 
professional educational organizations, for example, the six standards of the Interstate School 
Leaders Licensure Consortium (ISLLC) and the seven standards of the National Policy Board for 
Educational Administration (NPBEA, 2015).” 

Some states have personalized those national standards to their states such as: 

“ The William Cecil Golden School Leadership Development Program was established by the 
2006 Florida Legislature to provide a high quality, competency-based, customized, 
comprehensive and coordinated statewide professional development system for current and 
emerging school leaders. The program is aligned with and supports Florida’s Principal 
Leadership Standards , the standards of the National Staff Development Council, the Florida 
Professional Development Protocol Standards and NCLB requirements for high quality 
professional development (Florida Department of Education, 2015).” 

California has also established the California Professional Standards for Educational 
Leaders. “These standards were adapted from the Interstate School Leaders Licensure 
Consortium (ISLLC) Standards for School Leaders (1996). Washington, DC: Council of Chief 
State School Officers. Adaptations were made for the California Professional Standards for 
Educational Leaders (2001) by representatives from the California School Leadership Academy, 
as well as the Association of California School Administrators, California Commission on 
Teacher Credentialing, California Department of Education, and California colleges and 
universities. (California Department of Education, 2015).” Connecticut developed its own 
School Leadership Standards, which serve “...as the foundation for a variety of state functions, 
including leadership preparation program accreditation, licensure assessment, school 
administrator evaluation and professional development from induction through the professional 
certificate.. .by adapting the national Interstate School Leadership Licensure Consortium 
(ISLLC) Standards for use in Connecticut (Connecticut Department of Education, 2012).” 

Thus, the paucity of literature related to the perception of students’ impact upon 
educational leadership programs substantiates the importance of this study. Furthermore the 
CAEP requirements to “regularly and systematically assess perfonnance against its goals and 
relevant standards, track results over time, test innovations and the effects ... on subsequent 
progress and completion, and use results to improve program elements and processes” further 
demonstrates the significance of this study. 

Methodology 

This research study was conducted in July of 2015. The survey was administered to new students 
who will begin their educational leadership program this fall in one educational leadership 
program in the Commonwealth of Virginia. The analysis of the responses was conducted in late 
July 2015 in time for reporting during the NCPEA conference. 

Instrument 

The survey instrument being used in this study was created based on the 38 internship objectives 
developed by members of the faculty in the Educational Leadership Program at Virginia Tech. 
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The objectives were formulated through a review of the most recent ISLLC standards, as well as 
the standards outlined by the Virginia Department of Education (VDOE). Each of the internship 
objectives is aligned directly to one of the seven current ELCC standards as well as standards for 
the VDOE. 

The instrument has been divided into six distinct sections, with each section containing 
four statements for the students to respond to. The six broad sections are: 

1. Knowledge of leadership and change functions 

2. Knowledge of student services 

3. Knowledge of school operations 

4. Knowledge of school board policies 

5. Knowledge of human resource functions 

6. Knowledge of curriculum and instructional supervision 

Not all of the 38 internship objectives were included in the survey. Some were omitted as the 
objective was based on an activity rather than gamering specific knowledge. In total, 24 of the 
38 objectives were converted into survey items for this study. There is also a final section of the 
survey that allows students to make general comments regarding their knowledge levels, as well 
as any other area of interest or concern. 

Population 

Fifty-one students from five different Virginia Tech satellite sites across the Commonwealth 
were surveyed for this study. This represents approximately 92% of the total population. These 
sites include: Hampton Roads, Richmond, Northern Virginia, Abingdon, and Roanoke. Each 
student is about to begin his/her first term as a graduate student in the Principal Preparation 
Program within the Educational Leadership Program at Virginia Tech beginning in the fall of 
2015. All students are current practicing teachers, or central office personnel. Data from each 
site was analyzed independently, as well as collectively to provide information to individual 
faculty members, as well as the faculty as a whole. 

Data Collection and Statistical Analysis 

Data collection. Data was collected through the students’ completion of the Students' 
Perceptions of Knowledge Related to School Leadership survey via Qualtrics Survey 
Software™. A welcome email was sent to all participants during the second week of July 
informing them of the reason for the survey, as well as providing students a link to access the 
instrument. Students were asked to complete the survey as part of the program. The final date 
for collection of data was July 25, 2015. Data was then organized into an Excel spreadsheet 
entered into the Statistical Package for the Social Sciences (SPSS) for analysis. 

Validity. After the initial development of the survey in the spring of 2015, the assessment 
was then pilot-tested with several faculty members to help establish content and face validity. 
The feedback from the faculty was reviewed, and adjustments to the instrument were made as 
needed. 

The items and the instrument were next reviewed for content and face validity via an 
online focus group of current school and central office administrators in the Hampton Roads area 
during the month of May of 2015. Educational focus groups that provide feedback on an 
instrument help confirm its content validity since participants were practicing experts in the field 
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(Cannizzaro, 2007). Clause (2015) noted “content validity is most often measured by relying on 
the knowledge of people who are familiar with the construct being measured. These subject- 
matter experts are usually provided with access to the measurement tool and are asked to provide 
feedback on how well each question measures the construct in question.” As such, feedback 
from this group provided an effective method to establish both face and content validity, and 
resulted in additional adjustments to the instrument. One adjustment was to reorder several of the 
categories to improve the flow of the survey. Other adjustments were minor in nature, including 
words, punctuation and typos. 

Reliability. Rovai, et. al. (2012) recommends measuring internal consistency and 
reliability using Cronbach’s alpha. After the data collection has been completed, each of the six 
distinct categories based on the internship objectives was individually tested for reliability based 
on the responses to the four sub-questions under each category. As a final measure, the entire 
response set was tested to detennine overall reliability. Reliability tests resulting in an alpha of .7 
are generally accepted as having high reliability (Rovai, Baker & Ponton, 2012, p. 385). 
Cronbach’s alpha reliability coefficient generally ranges between 0 and 1. However, there is 
actually no lower limit to the coefficient. The closer Cronbach’s alpha coefficient is to 1.0 the 
greater the internal consistency of the items in the scale. George and Mallery (2003) provide the 
following rules of thumb: > .9 - Excellent, _ > .8 - Good, _ > .7 - Acceptable, _ > .6 - 

Questionable, _ > .5 - Poor, and_ < .5 - Unacceptable” (p. 231). 

Findings 

Knowledge of Leadership and Change Functions 

For responses to the intern’s performance related to Knowledge of leadership and change 
functions, 48 cases of the possible 51 were included in the analysis (Table 1). Students who did 
not have a complete data set were excluded. Cronbach’s alpha for the 48 of 51 items was .632 
(Table 2), which represents a questionable correlation between items. The instrument for 
Knowledge of leadership and change functions can be deemed somewhat reliable. 

Table 1 


Knowledge of leadership and change functions case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 


a. Listwise deletion based on all variables in the procedure. 


Table 2 

Knowledge of leadership and change functions _ 

Reliability statistics 

Cronbach’s Alpha a N 

Group 1 .632 4 
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Knowledge of Student Services 


For responses to the intern’s perfonnance related to Knowledge of student sendees, 48 cases of 
the possible 51 were included in the analysis (Table 3). Students who did not have a complete 
data set were excluded. Cronbach’s alpha for the 48 of 51 items was .804 (Table 4), which 
represents a good correlation between items. The instrument for Knowledge of student services 
can be deemed reliable. 

Table 3 


Knowledge of student services case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 

a. Listwise deletion based on all variables 

in the procedure. 

Table 4 



Knowledge of student services 



Reliability statistics 

Cronbach’s Alpha 

a 

N 

Group 1 

.804 

4 


Knowledge of School Operations 

For responses to the intern’s performance related to Knowledge of school operations, 48 cases of 
the possible 51 were included in the analysis (Table 5). Students who did not have a complete 
data set were excluded. Cronbach’s alpha for the 48 of 51 items was .850 (Table 6), which 
represents a good correlation between items. The instrument for Knowledge of school operations 
can be deemed reliable. 

Table 5 


Knowledge of school operations case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 

a. Listwise deletion based on all variables 

in the procedure. 

Table 6 



Knowledge of school operations 



Reliability statistics 

Cronbach’s Alpha 

a 

N 

Group 1 

.850 

4 
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Knowledge of School Board Policies 


For responses to the intern’s performance related to Knowledge of school board policies, 48 
cases of the possible 51 were included in the analysis (Table 7). Students who did not have a 
complete data set were excluded. Cronbach’s alpha for the 48 of 51 items was .896 (Table 8), 
which represents an excellent correlation between items. The instrument for Knowledge of 
school board policies can be deemed highly reliable. 

Table 7 


Knowledge of school board policies case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 

a. Listwise deletion based on all variables 

in the procedure. 

Table 8 



Knowledge of school board policies 


Reliability statistics 

Cronbach’s Alpha 

a 

N 

Group 1 

.896 

4 


Knowledge of Human Resource Functions 

Knowledge of human resource functions, 48 cases of the possible 51 were included in the 
analysis (Table 9). Students who did not have a complete data set were excluded. Cronbach’s 
alpha for the 48 of 51 items was .627 (Table 10), which represents a questionable correlation 
between items. The instrument for Knowledge of human resource functions can be deemed 
somewhat reliable. 

Table 9 


Knowledge of human resource functions case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 

a. Listwise deletion based on all variables 

in the procedure. 

Table 10 



Knowledge of human 

resource functions 


Reliability statistics 

Cronbach’s Alpha 

a 

N 

Group 1 

.627 

4 
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Knowledge of Curriculum and Instructional Supervision 

For responses to the intern’s performance related to Knowledge of curriculum and instructional 
supervision, 48 cases of the possible 51 were included in the analysis (Table 11). Students who 
did not have a complete data set were excluded. Cronbach’s alpha for the 48 of 51 items was 
.872 (Table 12), which represents an good correlation between items. The instrument for 
Knowledge of curriculum and instructional supervision can be deemed highly reliable. 

Table 11 


Knowledge of curriculum and instructional supervision case 
summary _ 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 


a. Listwise deletion based on all variables in the procedure. 


Table 12 

Knowledge of curriculum and instructional supervision 
Reliability statistics 

Cronbach’s Alpha a N 

Group 1 .872 4 


Reliability of Complete Survey 

For the final analysis conducted, the researchers tested all survey items combined to detennine 
the overall reliability of the instrument. 48 cases of the possible 51 were included in the 
analysis (Table 13). Students who did not have a complete data set were excluded. Cronbach’s 
alpha for the 48 of 51 items was .927 (Table 14), which represents an excellent correlation 
between items. The instrument overall can be deemed highly reliable. 

Table 13 


Overall instrument case summary 


Cases 

N 

% 

Valid 

48 

94.1 

Excluded 3 

3 

5.9 

Total 

51 

100.0 


a. Listwise deletion based on all variables in the procedure. 
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Table 14 


Overall instrument reliability 


Reliability statistics 

Cronbach’s Alpha 

a 

N 

Group 1 

.927 

24 


Conclusions and Future Study 

This research study has helped to determine that the instrument created for the purpose of 
detennining student perceptions of readiness at the beginning of the educational leadership 
program is both valid and reliable. Multiple measures were used to determine both face validity 
as well as content validity, including using a panel of experts in the field. The reliability of the 
instrument appears to be good overall. When the six independent groups were evaluated, Alpha 
ratings ranged from .627 (questionable) to .896 (excellent), but when the instrument was 
evaluated in totality it yielded an Alpha rating of .927 (excellent). For these reasons the 
researchers deem the instrument reliable. 

Recommendations for future study 

The researchers are aware that the n value used in this study is low, though it does represent the 
entire population of the study group at this time. It is recommended that this study be conducted 
again when the next cohort begins, and combine the two groups to increase the n value, thus 
improving the reliability. 

It is also recommended that the data collected through this research be used by the 
educational leadership faculty to make changes to their course content. A qualitative study could 
be conducted to detennine the level of changes made by the faculty to determine if the research 
is contributing to overall program improvement. 

Finally, it is recommended that the study population be given the survey again upon 
completion of the educational leadership program. This will allow the faculty to detennine the 
level of growth attained by students in the program in the areas of focus. Further changes to 
program content could be detennined after this additional study. 
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